Method and apparatus for processing pictures of mobile object

ABSTRACT

A feature amount of an inter-mobile unit relative movement are detected as an observation amount by an observation amount detecting section  26 , time series of the observation amounts are stored as an observation series into a storage section  27  to calculate a similarity of the observation series to a predetermined collision observation series by a classification section  28 . A determination section  29  determines to be a collision accident if, in a case where the similarity is larger than a predetermined value, a mobile unit associated with the similarity is at rest in a stoppage prohibition area set in a storage section  30  and another mobile unit is moving, and to be a mobile unit failure if collision determination conditions except for the similarity are met. By consisting of not only a first scalar obtained by quantizing a relative velocity vector between mobile units but also a second scalar obtained by quantizing a relative position vector between mobile units as the observation amount, a relative movement between mobile units is classified in more detail. A mobile unit is tracked in units of block by a mobile unit tracking section  25  to discriminate overlapped mobile units in pictures.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of application Ser. No. 10/120,374,filed Apr. 12, 2002, now pending.

This application is based upon and claims the priority of Japaneseapplication no. 2001-187502, filed Jun. 21, 2001, and U.S. patentapplication Ser. No. 10/120,374, filed Apr. 12, 2002, the contents beingincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method and an apparatus forprocessing pictures of mobile unit, more particularly, to a method andan apparatus for processing time-series pictures to detect an anomalysuch as collision or failure of mobile unit in the pictures.

2. Description of the Related Art

Early detection of a traffic accident can not only enhance a successrate in life saving by speedy rescue operation, but also alleviateaccident-related traffic congestion by speedup of the police inspectionat the site. Therefore, various kinds of automation in recognition oftraffic accident are expected.

In the publication of JP 2001-148019-A whose inventors are the same asthose of the present application, there is disclosed a mobile unitanomaly detection method processing time-series pictures to detect ananomaly of mobile unit in pictures, comprising the steps of:

Recently, a disc loading apparatus without a tray is widely used toreduce a size thereof and save a space. Additionally, a loadingapparatus has been disclosed in Korean Patent No. 10-0433415 filed bythe present applicant, which is able to selectively load discs indifferent sizes, for example, 80 mm and 120 mm.

(a) identifying mobile units in a frame picture at a time t on the basisof a correlation between frame pictures at times (t−1) and t;

-   -   (b) detecting a feature amount of a relative movement of a        second mobile unit relative to a first mobile unit as an        observation amount to store observation amounts in time-series        as an observation series;    -   (c) calculating a similarity of the observation series to each        reference series to classify a movement between mobile units;        and    -   (d) determining that a collision accident has occurred when the        similarity of the observation series to a collision reference        series is larger than a predetermined value.

According to this method, it is possible to automatically detect ananomaly such as a collision accident.

However, in a case where a camera angle is low with respect to a roadsurface, for example, if the second mobile unit approaches the firstmobile unit at rest and thereafter, the first mobile unit starts andstops, the second mobile unit overlaps the first mobile unit on picturesat times of the approach and a distance therebetween becomes zero, whichis sometimes wrongly determined as a time series pattern of a collisionaccident

Further, in the above publication, a scalar obtained by quantizingV/(d+ε) is used as an observation amount in the step (b), where Vdenotes a relative motion vector of the second mobile unit with respectto the first mobile unit and ε denotes a constant to avoid thedenominator to be zero.

By using this observation amount, various kinds of movements betweenmobile units can be classified with a small number of reference seriesbecause of the quantization.

However, there has been a problem of impossibility of more detailedclassification of movements between mobile units.

In the above publication, in the step (a), by using the identificationresult of a mobile unit in a frame picture at the time (t−1), the mobileunit in a frame picture at the time t can be identified with ease fromthe correlation.

However, in a case where mobile units are shot from the front thereof ata low camera angle with respect to a road surface in order to shoot awide area with one camera to track the mobile units, overlap betweenmobile units on a picture frequently occurs as shown in FIG. 13. At thetime (t−1), mobile units M1 and M2 are identified as one cluster withoutdiscriminating the mobile units M1 and M2 from each other. Although arepresentative motion vector of this cluster is used to identify thecluster including the mobile units M1 and M2 at the time t on the basisof the above-described correlation, accurate identification is disabledsince there is a difference in speed between the mobile units M1 and M2.At the next time (t+1), although the mobile unit M2 has been separatedfrom the mobile unit M1, the mobile units M2 and M3 are identified asone cluster since they overlap each other, disabling discrimination ofthe mobile units M2 and M3 from each other.

SUMMARY OF THE INVENTION

Accordingly, it is an object of the present invention to provide amethod and an apparatus for processing pictures of mobile unit, capableof automatically detecting anomaly such as a collision accident moreaccurately.

It is another object of the present invention to provide a method and anapparatus for processing pictures of mobile units, capable of performingmore detailed classification of various kinds of movements betweenmobile units with a small number of reference series.

It is still another object of the present invention to provide a methodand an apparatus for processing pictures of mobile units, capable ofidentifying different mobile units even if overlapping between themobile units frequently occurs.

In one aspect of the present invention, there is provided a mobile unitanomaly detection method of processing time series pictures to detect ananomaly of a mobile unit in a picture, comprising the steps of:

-   -   (a) detecting a feature amount of a relative movement of a        second mobile unit with respect to a first mobile unit as an        observation amount to store a time series of the observation        amounts as an observation series;    -   (b) calculating a similarity of the observation series to a        predetermined collision observation series; and    -   (c) when the similarity is larger than a predetermined value,        determining whether or not a collision accident has happened on        the basis of whether or not the first or second mobile unit is        at rest in a predetermined stoppage prohibition area.

In a case where mobile units approaching each other overlap in a pictureeven if the mobile units have not collided against each other because ofa low camera angle with respect to a road surface, collision accidentsare excessively detected when a collision accident is determined onlywith a similarity between an observation series and a reference series.However, according to this configuration, the collision accident betweenmobile units is determined more accurately, enabling reduction inexcessive detection of the collision accident.

In the above step (c), by considering whether or not another mobile unitis moving, chances of excessive detection of the collision accident canbe further reduced. When the similarity is smaller than thepredetermined value, determining whether or not the first or secondmobile unit at rest is in failure on the basis of whether the first orsecond mobile unit is at rest in the predetermined stoppage prohibitionarea.

Further, in a case where the same conditions for collision determinationare met except that the similarity obtained in the above step (b) issmaller than the predetermined value, it can be determined at a higherprobability that a mobile unit at rest is in failure. In this case aswell, by considering whether or not another mobile unit is moving, itcan be determined at a higher probability that a mobile unit at rest isin failure. In another aspect of the present invention, there isprovided an inter-mobile unit movement classification method ofprocessing time series pictures to classify a movement between mobileunits in at least one of the pictures, comprising the steps of:

-   -   (a) detecting a feature amount of a relative movement of a        second mobile unit with respect to a first mobile unit as an        observation amount to store a time series of the observation        amounts as an observation series;    -   (b) calculating a similarity of the observation series to a        reference series; and    -   (c) classifying a movement of the second object with respect to        the first mobile unit according to a value of the similarity;        wherein the observation amounts each include: a first scalar        obtained by quantizing an amount associated with both a relative        velocity V of the second mobile unit with respect to the first        mobile unit and a distance d between the first and second mobile        units; and a second scalar obtained by quantizing a relative        position vector of the second mobile unit with respect to the        first mobile unit.

With this configuration, since the second scalar obtained by quantizingthe relative position vector V is used, relative movements betweenmobile units can be classified that have been unable to be discriminatedfrom each other only with the first scalar associated with the relativevelocity vector, which enables grasping a situation more accurately,leading to a contribution to more accurate detection of a trafficaccident.

In still another aspect of the present invention, there is provided amobile unit identification method dividing each of time series picturesto blocks each including a plurality of pixels to process the pictures,wherein the method assigns identification code of a plurality of mobileunits included in a frame picture at a time t in units of the block, andobtains motion vectors of the plurality of mobile units in units of theblock, in a case where identification code of the plurality of mobileunits included in a frame picture at a time (t−1) have been assigned inunits of the block, and motion vectors of the plurality of mobile unitshave been obtained in units of the block, the method comprises the stepsof:

-   -   (a) moving a block j at the time (t−1), whose identification        code is IDj and whose motion vector is Vj, by the vector Vj to        obtain a substantially corresponding block i at the time t and        moving the block i by a vector −Vj to obtain a first box at the        time (t−1), to calculate an evaluation value associated with a        correlation between an image in the first box at the time (t−1)        and an image of the block i at the time t;    -   (b) moving a block k at the time (t−1), whose identification        code is IDk and whose motion vector is Vk, by the vector Vk to        obtain a substantially corresponding block which is the block i        at the time t and moving the block i by a vector −Vk to obtain a        second box at the time (t−1), to calculate an evaluation value        associated with a correlation between an image in the second box        at the time (t−1) and the image of the block i at the time t;        and    -   (c) assigning the identification code IDj or IDk to the block i        at the time t on the basis of magnitudes of the evaluation        values calculated in the steps (a) and (b).

With this configuration, since a motion vector of each block is used, itis possible to assign one of a plurality of identification codes to ablock within one cluster including a plurality of mobile units havingdifferent velocities at a time t; thereby enabling to divide the onecluster into clusters with different identification codes. That is, itbecomes possible to track mobile units, for which it was not possible inthe prior art, leading to a contribution to more accurate detection ofsuch as a collision accident or a traffic violation.

For example, the evaluation value of the step (a) includes a sum overp=1 to Na of a value associated with a value |VCm(t−1)−VBp(t−1)| on theassumption that an identification code of the block i at the time t isIDj, where VCm(t−1) denotes a motion vector of a block m at the time(t−1), the block m at the time (t−1) corresponds to the block i at thetime t, and VBp(t−1) denotes a motion vector of a block whoseidentification code is IDj and which is adjacent to the block m at thetime (t−1), wherein the evaluation value of the step (b) includes a sumover q=1 to Nb of a value associated with a value |VCn(t−1)−VBq(t−1)|(for example, |VCn(t−1)−VBq(t−1)|^(r), r>1) on the assumption that anidentification code of the block i at the time t is IDk, where VCn(t−1)denotes a motion vector of a block n at the time (t−1), the block n atthe time (t−1) corresponds to the block i at the time t, and VBq(t−1)denotes a motion vector of a block whose identification code is IDkwhich is adjacent to the block n at the time (t−1).

With this configuration, even if error of a motion vector at the time(t−1) is large because almost the same pixels distribute, it becomespossible to assign block identification codes more accurately, resultingin contribution to more accurate detection of such as a collisionaccident or a traffic violation.

Other aspects, objects, and the advantages of the present invention willbecome apparent from the following detailed description taken inconnection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram showing an intersection and an apparatus,placed at the intersection, of an embodiment according to the presentinvention;

FIG. 2 is a functional block diagram of the mobile unit anomalydetection apparatus in FIG. 1;

FIG. 3 is a flow diagram showing a processing of the determinationsection in FIG. 2;

FIG. 4 is an illustration of slits set at 4 entrances to theintersection and 4 exits therefrom and identification codes of mobileunits assigned to blocks;

FIG. 5 is an illustration of vectors for explaining a processing at theobservation amount detecting section in FIG. 2;

FIG. 6 is an illustration of quantization of a vector V/D;

FIG. 7 is an illustration of quantization of a relative position vectorand time-series classification;

FIGS. 8(A) and 8(B) are both illustrations of classification ofmovements between mobile units that pass close by each other;

FIGS. 9(A) and 9(B) are both illustrations of classification ofmovements between mobile units that pass close by each other;

FIG. 10 is an illustration of a time-series pattern in a collisionaccident

FIG. 11 is an illustration of determination of a mobile unit collisionaccident at an intersection;

FIG. 12 is an illustration of determining a failure of a mobile unit atan intersection;

FIG. 13 is an illustration of a case where an overlap between mobileunits on a picture frequently occurs;

FIG. 14 is an illustration of preparing an object map;

FIG. 15 is an illustration of preparing an object map;

FIG. 16 is an illustration of preparing an object map;

FIG. 17 is an illustration of preparing an object map;

FIG. 18 is an illustration of preparing an object map; and

FIG. 19 is an illustration of preparing an object map.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring now to the drawings, wherein like reference charactersdesignate like or corresponding parts throughout several views,preferred embodiment of the present invention is described below.

FIG. 1 is a schematic diagram showing an intersection and a mobile unitanomaly detection apparatus, placed at the intersection, of anembodiment according to the present invention.

This apparatus is provided with an electronic camera 10 shooting theintersection to output a picture signal and a mobile unit anomalydetection apparatus 20 processing pictures to automatically detect acollision accident between mobile units and a mobile unit failure.

FIG. 2 is a functional block diagram of the mobile unit anomalydetection apparatus 20 in FIG. 1. Of constituents of the mobile unitanomaly detection apparatus 20, each constituent other than storagesections can also be constructed of computer software or a dedicatedhardware. Time series pictures shot by the electronic camera 10 arestored into an image memory 21 at a rate of, for example, 12 frames/sec.

A background picture generation section 22 is provided with a storagesection and a processing section. This processing section accesses theimage memory 21 to prepare histograms of respective pixels, eachhistogram having corresponding pixel values of all the frames, forexample, over the past 10 minutes to generate a picture with no mobileunit therein as a background picture whose each pixel value is the modeof the corresponding histogram, and to store the background picture intothe storage section. This processing is repeated periodically to updatethe background picture.

As shown in FIG. 4, data of the positions and sizes, in a picture frame,of slits EN1 to EN4 disposed at 4 entrances to an intersection and EX1to EX4 disposed at 4 exits therefrom are in advance set in an IDgeneration/deletion section 23. The ID generation/deletion section 23reads picture data in the entrance slits EN1 to EN4 from the picturememory 21 to determine whether or not a mobile unit exists in theentrance slits in block units. Squares in a mesh of FIG. 4 are blocks,each block is of a size of, for example, 8×8 pixels and in a case whereone frame is constituted of 480×640 pixels, one frame is divided into60×80 blocks. Whether or not a mobile unit exists in a block isdetermined by whether or not a total sum of differences between pixelsin the block and corresponding pixels of the background picture isgreater than a predetermined value. The determination is also performedin a mobile unit tracking section 25.

The ID generation/deletion section 23 assigns a new clusteridentification codes ID to a block when determined that a mobile unitexists in the block. when determined that a mobile unit exists in ablock adjacent to another block to which ID has been assigned, IDgeneration/deletion section 23 assigns the same ID as that of the blockhaving been assigned to this adjacent block. This block to which ID hasbeen assigned may be one adjacent to an entrance slit. For example inFIG. 4, ID=1 is assigned to blocks in the entrance slit EN1 in which amobile unit exists and ID=5 is assigned to blocks in the entrance slitEN4 □and their neighboring blocks□ in which a mobile unit exists.

Assignment of ID is performed to corresponding blocks in an object mapstorage section 24. The object map storage section 24 is used forstoring information (object map) for facilitation of processing inregard each of the blocks 60×80 in the above case, and the informationincludes flags each indicating whether or not ID has been assigned. Inregard to each block, when the ID has been assigned, the informationfurther includes the ID number and a block motion vector describedlater. Note that without using the flag, ID=0 may be used for indicatingno assignment of ID. Further, the most significant bit of ID may be theflag.

For each cluster having passed through an entrance slit, the mobile unittracking section 25 assigns the same ID to blocks located in a movingdirection and deletes the same ID of blocks located in a directionopposite to the movement, that is, performs tracking processing forclusters. The mobile unit tracking section 25, as described later,generates an object map at a time t on the basis of an object map andframe picture at a time (t−1).

The mobile unit tracking section 25 performs the tracking processing asfar as and within an exist slit for each cluster.

The ID generation/deletion section 23 further checks whether or not IDis assigned to the blocks in the exit slits EX1 to EX4 on the basis ofcontents of the object map storage section 24 and if assigned, deletesthe ID when the cluster having the ID has passed though an exit slit.For example, when transition has been performed from a state where an IDis assigned to blocks in the exit slit EX1 in FIG. 4 to a state where noID is assigned thereto, ID=3 is deleted. The deleted ID can be used asthe next ID to be generated.

An observation amount detecting section 26 obtains a mean motion vectorof each cluster on the basis of contents of the object map storagesection 24 as a motion vector of the cluster, obtains a relative motionvector and a relative position vector between clusters, further obtainsthe shortest distance between the clusters, and thereby obtains anobservation amount described below. The observation amount detectingsection 26 stores the observation amount into an observation seriesstorage section 27 to generate an observation series in regard to eachbetween clusters.

As an observation amount, consideration is given to a first scalarobtained by quantizing an amount associated with a relative motionvector and a second scalar obtained by quantizing a relative positionvector. First, description will be given of the first scalar and itstime series.

For example, FIG. 5(B) shows a relative motion vector V of a motionvector V1 of a mobile unit M1 and a motion vector V2 of a mobile unit M2shown in FIG. 5(A). As shown in FIG. 5(A), letting d be the shortestdistance between the mobile units M1 and M2, consider a vector V/D,where D=d+ε and ε is a constant for guaranteeing D>0 when d=0. It willbe easy to determine a collision from a time series of vectors V/D sinceV=0, namely V/D=0 in collision and |V/D| takes large values before andafter the collision. In order to simply classify many of relative statesbetween the mobile units M1 and M2, the vector V/D is quantized to ascalar as shown in FIG. 6. That is, letting an area divide into regionsas shown in FIG. 6 with the origin of the vector V/D being the center ofthe area, and assigning scalars to respective divided regions, thevector V/D is quantized to the scalar of a region to which the tip ofthe vector belongs.

For example, in a case where a time series of V/D=v is v0→v1→v2, whichis denoted by (v0, v1, v2) and quantized to (0, 1, 2). The time seriesof quantized observation amounts is referred to as an observationseries. In a collision accident shown in FIG. 10, the observation seriesover a time t=4 to 9 is denoted by (1, 2, 3, 0, 8, 7). By suchquantization, various kinds of collision accidents can be easilyrecognized since a collision pattern and another collision patternanalogous to it have the same observation series as each other. Further,by the quantization, an observation series is a simple sequence ofnumerical values, therefore a subsequent recognition process can besimplified.

Determination of a collision accident or not is performed by whether ornot a similarity between an observation series and a reference series ofeach of some collision accidents (for example, observation series ofeach collision accident actually having happened) exceeds apredetermined value. Although the number of necessary reference seriescan be reduced by the above quantization, in order to further reduce thenumber, in a case where a time series pattern of a relative movementbecomes the same by rotating a stationary coordinate system, anobservation series is adjusted to be the same. For this purpose,regarding the above (v0, v1, v2) for example, the reference direction ofthe first vector v0 in the time series, that is, an angle θ to the Xaxis of FIG. 6, is obtained, followed by a rotation of −θ of the vectorsv0, v1 and v2. FIG. 5(C) shows the rotation in a case where V/D is thefirst vector in a time series. by the rotation, an observation amount ofthe relative vector V/D of the mobile unit M2 with respect to the mobileunit M1 and an observation amount of the relative vector −V/D of themobile unit M1 with respect to the mobile unit M2 are equal to eachother.

In a case where a mobile unit M1 and a mobile unit M2 spaced apart fromeach other approach and pass close by each other, thereafter they arespaced apart from each other, which is one of patterns of FIGS. 8(A),8(B), 9(A) and 9(B) and doted lines therein show tracks of the mobileunits. Observation series in the respective cases are as follows forexample:

-   -   FIG. 8(A):{1, 1, 2, 2, 3, 1, 1, 1, 1, 1, 1}    -   FIG. 8(B):{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}    -   FIG. 9(A):{1, 1, 2, 2, 3, 1, 1, 1, 1, 1, 1}    -   FIG. 9(B):{1, 1, 2, 2, 3, 1, 1, 1, 1, 1, 1}

As is apparent from these observation series, it is not possible toidentify these relative movements from each other by the observationseries of FIGS. 8(A), 8(B), 9(A) and 9(B). Therefore, in order to enableto identify the relative movements, a relative position vector of themobile unit M2 with respect to the mobile unit M1 is quantized to obtainthe second scalar as shown in FIG. 7. That is, an area is divided intoregions with dotted lines as shown in FIG. 7, wherein the center of thearea is the origin of the relative position vector P12 and a referencedirection is equal to a direction of the motion vector V1 of the mobileunit M1, and scalars are assigned to the respective divided regions. Therelative position vector P12 is quantized as a scalar of a region towhich the tip thereof belongs.

In a case of FIG. 8(A) where mobile units M1 and M2 are moving inopposite directions with passing close by each other, letting a relativeposition vector P12 at the time t be P12(t), a time series of relativeposition vectors P12(t−1), P12(t) and P12(t+1) of the mobile unit M2with respect to the mobile unit M1 are quantized as {20, 80, 60}.

The reason why the first scalar is represented by a one-digit numericalvalue while the second scalar by a two-digit numerical value with thelower digit being 0 is that both can be easily synthesized. For example,in a case where the first scalar is 6 and the second scalar is 20, avalue of 26, which is the sum of both, represents an observation amount.In a case where the observation amount is defined as the sum of thefirst scalar, and the second scalar with respect to the mobile unit M2,the observation series of FIGS. 8(A), 8(B), 9(A) and 9(B) arerepresented as follows:

-   -   FIG. 8(A):{21, 21, 82, 82, 83, 81, 81, 81, 81, 61, 61}    -   FIG. 8(B):{21, 21, 41, 41, 41, 41, 41, 41, 41, 61, 61}    -   FIG. 9(A):{61, 61, 42, 42, 43, 41, 41, 41, 41, 21, 21}    -   FIG. 9(B):{21, 21, 42, 42, 43, 41, 41, 41, 41, 61, 61}

In a case where the observation amount is defined as the sum of thefirst scalar, and the second scalar with respect to the mobile unit M1,the observation series of FIGS. 8(A), 8(B), 9(A) and 9(B) arerepresented as follows:

-   -   FIG. 8(A):{21, 21, 82, 82, 83, 81, 81, 81, 81, 61, 61}    -   FIG. 8(B):{21, 21, 41, 41, 41, 41, 41, 41, 41, 61, 61}    -   FIG. 9(A):{21, 21, 82, 82, 83, 81, 81, 81, 81, 61, 61}    -   FIG. 9(B):{61, 61, 82, 82, 83, 81, 81, 81, 81, 21, 21}

As shown in FIG. 7 with straight lines each having an arrow, a casewhere the second scalar changes as 20→80→60 is classified as PAS0, acase where the second scalar changes as 20→40→60 is classified as PAS1,a case where the second scalar changes as 60→40→20 is classified asPAS2, and a case where the second scalar changes as 60→80→20 isclassified as PAS3. Further, in a case where a change in relativeposition vector of the mobile unit M2 with respect to the mobile unit M1is represented as PASk and a change in relative position vector of themobile unit M1 with respect to the mobile unit M2 is represented asPASm, it is denoted by PASkm.

In each case of FIGS. 8(A) and 8(B), a change in relative positionvector of the mobile unit M2 with respect to the mobile unit M1 and achange in relative position vector of the mobile unit M1 with respect tothe mobile unit M2 are the same as each other, which are classified asPAS00 and PAS11, respectively. On the other hand, in cases of FIGS. 9(A)and 9(B), a change in relative position vector of the mobile unit M2with respect to the mobile unit M1 and a change in relative positionvector of the mobile unit M1 with respect to the mobile unit M2 differfrom each other, which are classified as PAS20 and PAS13, respectively.

In such a way, although it is not possible to discriminate and recognizethe relative movement of mobile units only using the first scalar, thisis enabled by classifying the relative movement using the second scalar,which enables more accurate understanding of the states.

For example, letting the number of clusters on an object map be 3, andIDs be 1, 2 and 3, and an observation series between clusters of ID=iand ID=j is denoted by OSi,j, observation series OS1,2; OS2,1; OS2,3;OS3,1 and OS1,3 are stored in the observation series storage section 27of FIG. 2. The reason why OSi,j and OSj,i are both stored is to enableclassification of the above PASkm with focusing attention on secondscalars in a time series. In the observation series storage section 27,each of stored observation series is consisted of a predeterminednumber, for example, 24, of observation amounts, and every time advanceby 1 the most oldest observation amount is deleted and a new observationamount is added in each observation amount.

A classification section 28 reads observation series between clustersstored in the observation series storage section 27, to calculatesimilarities between each of the observation series and each ofpredetermined reference series of collision accidents and others and togive the calculated similarities to a determination section 29 asclassification results. Each of the observation series is a time seriesof observation amounts each having a combination of the first scalar andthe second scalar, and the reference series are constituted in a similarmanner. Note that calculation may be performed on a similarity to eachpredetermined reference series of a collision accident only with respectto the first scalar of each observation series and on a similarity toeach predetermined reference series of an above PASkm with respect tothe second scalar thereof to give the determination section 29 thecalculated similarities as classification results.

Known pattern similarity calculation methods such as a hidden Markovmodel (HMM) method and a pattern matching method can be applied. Forexample, as disclosed in the above-described publication of JP2001-148019-A, a probability of occurrence of the observation series maybe calculated as a similarity by the HMM whose parameters have beendetermined using observation series of actual collision accidents aslearning series.

Now, description will be given of detection of a collision accident anda mobile unit failure with reference to FIGS. 11 and 12.

In FIG. 11, assume that there has been occurred a collision sequencebetween mobile units M1 and M2 as shown in FIG. 10 or a non-collisionsequence similar to this collision sequence and the mobile units M1 andM2 is stopping. This collision sequence is similar to that of a casewhere a mobile unit M3 is at rest to make its right turn and at thattime, a mobile unit M4 approaches the mobile unit M3 from the rear, andtherefore the mobile unit M3 moves a bit forward and stops. In a casewhere mobile units are shot at a low camera angle with respect to a roadsurface, the mobile units look like overlapping to each other even whenthe mobile units are actually spaced apart from each other, so a chancearises where the situation is classified as collision between the mobileunits in pictures.

However, the mobile units M3 and M4 are in an area where stoppage ispermitted for a right turn. Contrast to this, mobile units M1 and M2exist in an area where stopping is prohibited; therefore, there is ahigh possibility that a collision has happened between the mobile unitsM1 and M2. In this case, there is a possibility that the mobile units M1and M2 are at rest since an ambulance or the like is approaching theintersection. Therefore, if there exists another mobile unit passingclose by the mobile units M1 or M2 at rest, a possibility of collisionbetween the mobile units M1 and M2 becomes higher.

From the above consideration, a determination is performed that the caseis collision between the mobile units M1 and M2: if

-   -   (1) a relative movement between the mobile units M1 and M2 is        classified as collision;    -   (2) the mobile unit M1 or M2 is at rest in a stoppage        prohibition area; and    -   (3) a mobile unit other than the mobile units M1 and M2 are        moving in the intersection.

Further, as shown in FIG. 12, in a case where the mobile unit M1 has notcollided with another car, but is at rest in the stoppage prohibitionarea, there is a high possibility that the mobile unit M1 is in failure.In this case, if another mobile unit is moving in the intersection,there is a higher possibility that the mobile unit M1 is in failure.Therefore, if the above (1) is determined negative but the above (2) and(3) are determined positive, it is determined that the mobile unit atrest is in failure.

FIG. 3 is a flow diagram showing a processing in the determinationsection 29 of FIG. 2.

(S1) The determination section 29 reads a maximum collision similarityCS of a pair of clusters from the classification section 28.

(S2 to S4) If the collision similarity CS is larger than a predeterminedvalue CS0, then a flag F is set, or else the flag F is reset.

(S5) An observation series corresponding to the pair of the clusters inthe step S1, is read from the observation series storage section 27, andif the first scalars of a predetermined number of observation amountsfrom the latest time are all zero, that is, if a state where a relativemotion vector V=0 lasts for over a predetermined time period, then theobject map storage section 24 is referred to investigate whether or notmotion vectors of all blocks belonging to the clusters having the IDsare zero, as a result of which if all the motion vectors are zero, it isdetermined that the mobile unit is at rest. If not at rest, then theprocess returns to the step S1 to read a maximum CS of another pair ofclusters, or else the process advances to the step S6.

(S6) It is investigated whether or not the cluster at rest exists withinany one of the stoppage prohibition areas stored in advance in astoppage prohibition area storage section 30, and if exists, then theprocess advances to step S7, or else the process returns to the step S1.

(S7) In the classification results of the classification section 28, ifa similarity of a pair of clusters one of which is other than theclusters at the step S1 to a reference series of anyone of the PAS00,PAS11, PAS20 or PAS13 exceeds a predetermined value, then it isdetermined that there exists a mobile unit moving in the intersectionand the process advances to step S8, or else the process returns to thestep S1.

(S8 to S10) If F=1, then it is determined to be an accident, or if F=0,then it is determined that the mobile unit is in failure, and thisresult is outputted. Then the process returns to step S1.

According to such processing, a collision accident and a mobile unitfailure can be automatically detected with a high probability.

Now, detailed description will be given of a method for preparing anobject map at a time t on the basis of an object map and a frame pictureat a time (t−1) and a frame picture at the time t in the mobile unittracking section 25 of FIG. 2.

In a case where a mobile unit is shot from the front thereof at a lowcamera angle with respect to a road surface in order to shoot a widearea with one camera to track mobile units, overlaps between mobileunits in pictures frequently occurs as shown in (A) to (C) of FIG. 13.

FIGS. 14 and 15 show enlarged pictures of (A) and (B) of FIG. 13,respectively. Dotted lines are for dividing a picture into blocks. InFIG. 14, overlapped mobile units correspond to one cluster C12 in theobject map storage section 24 of FIG. 2, and assume that the mobileunits M1 and M2 are not yet discriminated from each other. On the otherhand, a cluster 3 corresponds to one mobile unit M3.

A block on the i-th row and the j-th column at a time t is denoted byB(t: i, j). As shown in FIG. 14, motion vectors of blocks B (t−1:11, 13)and B (t−1:14, 13) are denoted by V2 and V3, respectively. The tips ofthe motion vectors V2 and V3 both exist in a block (t−1:18, 11). In apicture at the time t of FIG. 15, frames SB2 and SB3 correspond torespective regions where the blocks B (t−1:11, 13) and B (t−1:14, 13) inthe picture of FIG. 14 are moved by the motion vectors V2 and V3,respectively.

Next, the motion vectors V2 and V3 are translated such that the tips ofthe motion vectors V2 and V3 both coincide with the center of the blockB (18, 11). Then the motion vectors V2 and V3 are inversed and the blockB (t−1:18, 11) which is hatched is moved by vectors −V2 and −V3 toobtain frames SBR2 and SBR3, respectively, as shown in FIG. 16. Imagesin the boxes SBR2 and SBR3 are estimated ones on the assumption that animage in the block B (t: 18, 11) of FIG. 17 would have belonged to theclusters C12 and C3 of FIG. 16 at the time (t−1). IDs of the clustersC12 and C13 are denoted by ID12 and ID3, respectively.

An evaluation value UD associated with a similarity between the image inthe box SBR2 of FIG. 16 and the image in the block B(t: 18, 11) iscalculated with the following equation, and the value is denoted by UD(ID12).UD=Σ|SP(t−1:i, j)−BP(t: i, j)|  (1)

-   -   where SP(t−1: i, j) and BP(t: i, j) denote pixel values on the        i-th row and the j-th column in the box SBR2 of FIG. 16 and in        the block B(t: 18, 11) of FIG. 17, respectively, and Σ denotes a        sum over i=1 to 8 and j=1 to 8 (a sum over all pixels in a block        or box). The smaller the evaluation value UD is, the higher the        correlation is.

Likewise, an evaluation value UD associated with a correlation betweenthe image in the block SBR3 of FIG. 16 and the image in the block B(t:18, 11) of FIG. 17 is calculated, and the value is denoted by UD(ID3).

In the case of FIGS. 16 and 17, UD(ID3)<UD(ID12) holds and thereby ID3is assigned to the block B(t: 18, 11).

In such a way, by using a motion vector of each block, different IDs canbe assigned to blocks included in the cluster C123 including a pluralityof mobile units at the time t, and thereby one cluster C123 can bedivided into subclusters with different IDs.

How to find out the block B(t−1:11, 13) in the cluster C12 and the blockB(t−1: 14, 13) in the cluster C3 both corresponding to the block B(t:18, 11) belonging to the cluster C123 of FIG. 15 is as follows: That is,letting a vector from the center of the block B(t−1: i, j) to the centerof the block B(t−1:18, 11) be V(18−j, 11−j) and a motion vector of theblock B (t−11: i, j) be V(t−1: i, j), it is equivalent to find out ablock B(t−1: i, j) having V(t−1: j, j) satisfying the followingexpression:|V(18−i, 11−j)−V(t−1:i, j)|<ΔV

-   -   where ΔV is a constant whose value is for example, three times        the number of pixels on one side of a block. In a case where a        plurality of blocks corresponding to the block B(t: 18, 11)        exist in the cluster C12 or in a case where a plurality of        blocks corresponding to the block B(t: 18, 11) exist in the        cluster C3, the evaluation value is obtained for each of such        blocks and ID corresponding to the least evaluation value is        assigned to the block B(t: 18, 11).

The above procedure is applied to other blocks belonging to the clusterC123 of FIG. 15 in a similar manner.

In the above case where ID3 is assigned to the block B(t: 18, 11), amotion vector of the block can be estimated to be almost equal to thevector V3. In order to obtain the motion vector of the block B(t: 18,11) more accurately, the box SBR3 is shifted by one pixel at a time in apredetermined range whose center is coincident with that of the boxSBR3, the evaluation value is obtained for every shift, and the motionvector of the block B(t: 18, 11) is determined to be a vector whoseorigin is the center of the box SBR3 when the evaluation value assumesthe minimum (the highest correlation) and whose tip is the center of theblock B(t: 18, 11). A motion vector of a block at a time t is determinedby such a block matching each time when ID is assigned to the block.

In order to estimate a similarity more accurately, amounts describedbelow are considered.

Part of the box SBR3 in FIG. 16 is outside the cluster 3 and as theoutside area is wider, it can be considered that a probability that IDof the block B(t: 18, 11) of FIG. 17 is ID3 is low. Therefore, assumingthat ID of the block B(t: 18, 11) is equal to ID3, the number S(t−1) ofpixels in the box SBR3 and belonging to the cluster C3 is obtained, andan evaluation value U associated with a correlation between the image inthe box SBR3 of FIG. 16 and the image in the block B(t: 18, 11) of FIG.17 is calculated with the following equation, and the calculated valueis denoted by US(ID3):US=(S(t−1)−64)²  (2)

The smaller the evaluation value UD is, the higher the correlation is.Likewise, assuming that ID of the block B(t: 18, 11) is equal to ID12and the number S of pixels in the box SBR2 and belonging to the clusterC12 is obtained to calculate the evaluation value US, and the value isdenoted by US(ID12). In cases of FIGS. 16 and 17, US(ID12)=0 andUS(ID3)>US(ID12) hold.

U=αUD+βUS which is a linear combination of the above equations (1) and(2) is defined as an evaluation function, and it is determined that thesmaller the evaluation value U, the higher the similarity is, where αand β are positive constants and determined on the basis of practicalexperiences such that the evaluation of similarity becomes more correct.

For each block of FIG. 17 at the time t, it is determined whether ID12or ID3 is assigned to in a similar way as described above. Since aprobability of a wrong determination is high in a case where theabsolute value of a difference of evaluation values |U(ID12)−U(ID3)| isequal to or less than a predetermined value, no ID is assigned and thefollowing amount is considered with respect to an image at a time t. Forexample, in a case where it is assumed that ID of the block B(t: 18, 11)of FIG. 17, which has not been determined, is equal to ID3, the numberN(t) of blocks assigned with ID3 among 8 blocks adjacent to the blockB(t: 18, 11) is obtained, an evaluation value UN is calculated with thefollowing equation, and the value is denoted by UN(ID3):UN=(N(t)−8)².  (3)

The smaller the evaluation value, the higher the correlation is.Similarly, in a case where it is assumed that ID of the block B(t: 18,11) of FIG. 17 is equal to ID12, the number N of blocks assigned withID12 among 8 blocks adjacent to the block B(t: 18, 11) is obtained, theevaluation value UN is calculated, and the value is denoted by UN(ID12).

Further, when the error of a motion vector at a time (t−1) obtained byblock matching is large since almost the same pixel values aredistributed, a case can be considered where the absolute value|U(ID12)−U(ID3)| of a difference in evaluation values of the linearcombination U=αUD+βUS of the above equations (1) to (3) is equal to orless than a predetermined value. Therefore, by paying attention tomotion vectors of blocks in the neighborhood of blocks B(t−1:14, 13) andB(t−1:11, 13) corresponding to the block (t: 18, 11), an evaluation of asimilarity is made more correct. That is, an evaluation value UV iscalculated using the following equation which includes a motion vectorVC(t−1)=V3 of the block B(t−1:14, 13) at the time (t−1) corresponding tothe B(t: 18, 11) on the assumption that the block B(t: 18, 11) has ID3;and motion vectors VBi(t−1), for i=1 to NX and NX=NX3, of blocks (blockseach attached with a small black dot in FIG. 16) which are ones among 8blocks adjacent to the block B(t−1:14, 13) and whose ID is equal to ID3,and the evaluation value is denoted by UV(ID3):UV=Σ|VC(t−1)−VBi(t−1)|² /NX  (4)

-   -   where Σ denotes a sum over i=1 to NX. The smaller the evaluation        value is, the higher the correlation is. Similarly, an        evaluation value UV is calculated in regard to a motion vector        VC=V2 of the block B(t−1:11, 13) at the time (t−1) corresponding        to the B(t: 18, 11) on the assumption that the block B(t:        18, 11) has ID12; and motion vectors VBj(t−1), for j=1 to NX and        NX=NX12, of blocks (blocks each attached with a mark x in        FIG. 16) which are ones among 8 blocks adjacent to the block        B(t−1:11, 13) and whose ID is equal to ID12, and the evaluation        value is denoted by UV(ID12).

A linear combination of the above equations (1) to (4),U=αUD+βUS+γUN+ΔUV  (5)is used as an evaluation function and it is determined that the smallerthe evaluation value, the higher the similarity is, where γ and δ arealso positive constants and determined on the basis of practicalexperiences such that the evaluation of similarity becomes more correct.

In such a way, not only is it determined whether ID is ID12 or ID3 ineach block in the cluster C123 of FIG. 17, but a motion vector of eachblock is also determined. That is, an object map at a time t isdetermined, and as shown in FIG. 18, even if the mobile unit M2 overlapsboth of the mobile units M1 and M3, the cluster can be divided intoclusters with different IDs.

Similarly, an object map at the time (t+1) can be obtained from a framepicture and an object map at the time t. Since C12 and C3 arediscriminated from each other at the time t and the mobile unit M1 isseparated from the mobile unit M2 in the frame picture at the time(t+1), as shown in FIG. 19, C1 to C3 corresponding to the mobile unitsM1 to M3, are discriminated from each other at the time (t+1).

Note that in a case where the equation (5) is used, one or more of β, γand δ may be zero in order to reduce a calculation time.

Further, an object map X at a time (t−1) may be copied into a work areaas an object map Y prior to preparation of an object map at a time t anda motion vector Vi of each block i whose ID is equal to IDα in theobject map X may be replaced with a mean vector (ΣΔVj)/p, for j=1 to p,where ΔV1 to ΔVp are motion vectors of all blocks, in the object map Y,including a block corresponding to the block i and blocks adjacent tothis corresponding block and having ID=IDα. With such a procedure, in acase where errors of motion vectors are large since a texture in a blockis similar to those in adjacent blocks, the errors are reduced. Copyingto a work area is for uniquely determining the mean vector.

Then, description will be given of a method for obtaining an object mapmore accurately using the evaluation function of the above equation (5)and an object map at a time t obtained as described above as an initialcondition. Since this method itself is the same as disclosed in theabove publication except for the evaluation function U, an outlinethereof will be given.

The blocks in the clusters C12 and C3 of FIG. 18 are denoted by BKi, fori=1 to n. ID of a block BKi is a variable IDi and the U thereof isexpressed as U(BKi, IDi). IDi is equal to ID12 or ID3. ID1 to IDn aredetermined such that a sum UT of evaluation values U over i=1 to n,UT=ΣU(BKi, IDi),takes the minimum. Initial values of ID1 to IDn are given by an objectmap at the time t obtained as described above.

Although preferred embodiment of the present invention has beendescribed, it is to be understood that the invention is not limitedthereto and that various changes and modifications may be made withoutdeparting from the spirit and scope of the invention.

For example, the apparatus of FIG. 2 automatically detecting a collisionaccident and a vehicle failure has characteristics in the determinationsection 29 and observation amounts detected by the observation amountdetecting section 26 may be only of the first scalars. Further, aconfiguration in which an observed value including the first and secondscalars are used can be applied for use in an apparatus performing otherthan determination of a collision accident and a vehicle failure, forexample, in an apparatus automatically classifying movements of mobileunits to get a statistics. Furthermore, a processing method in themobile unit tracking section 25 using the above-described evaluationfunction U can be applied to various kinds of mobile unit trackingapparatuses.

1. A mobile unit identification method dividing each of time seriespictures to blocks each including a plurality of pixels to process saidpictures, wherein said method assigns identification codes of aplurality of mobile units included in a frame picture at a time t inunits of the block, and obtains motion vectors of said plurality ofmobile units in units of the block, in a case where identification codesof said plurality of mobile units included in a frame picture at a time(t−1) have been assigned in units of the block, and motion vectors ofsaid plurality of mobile units have been obtained in units of the block,said method comprises: (a) moving a block j at the time (t−1), whoseidentification code is IDj and whose motion vector is Vj, by the vectorVj to obtain a substantially corresponding block i at the time t andmoving said block i by a vector −Vj to obtain a first box at the time(t−1), to calculate an evaluation value associated with a correlationbetween an image in said first box at the time (t−1) and an image ofsaid block i at the time t; (b) moving a block k at the time (t−1),whose identification code is IDk and whose motion vector is Vk, by thevector Vk to obtain a substantially corresponding block which is saidblock i at the time t and moving said block i by a vector −Vk to obtaina second box at the time (t−1), to calculate an evaluation valueassociated with a correlation between an image in said second box at thetime (t−1) and said image of said block i at the time t; and (c)assigning said IDj or said IDk to said block i at the time t on thebasis of magnitudes of said evaluation values calculated in the steps(a) and (b).
 2. The mobile unit identification method according to claim1, wherein said evaluation value of said step (a) includes a sum overp=1 to Na of a value associated with a value |VCm(t−1)−VBp(t−1)| on theassumption that an identification code of said block i at the time t isIDj, where VCm(t−1) denotes a motion vector of a block m at the time(t−1), said block m at the time (t−1) corresponds to said block i at thetime t, and VBp(t−1) denotes a motion vector of a block whoseidentification code is IDj and which is adjacent to said block m at thetime (t−1), wherein said evaluation value of said step (b) includes asum over q=1 to Nb of a value associated with a value|VCn(t−1)−VBq(t−1)| on the assumption that an identification code ofsaid block i at the time t is IDk, where VCn(t−1) denotes a motionvector of a block n at the time (t−1), said block n at the time (t−1)corresponds to said block i at the time t, and VBq(t−1) denotes a motionvector of a block whose identification code is IDk and which is adjacentto said block n at the time (t−1).
 3. The mobile unit identificationmethod according to claim 1, further comprising the step of: prior toprocessing on said frame picture at the time t, in said frame picture atthe time (t−1), replacing a motion vector of each block having anassigned identification code IDx with a mean motion vector obtained byaveraging motion vectors of this block and blocks adjacent to this blockand having said identification code IDx.
 4. A mobile unit identificationapparatus comprising: a picture storage device storing time seriespictures; and a picture processing device, dividing each of said timeseries pictures stored to blocks each including a plurality of pixels toprocess said pictures, assigning identification code of a plurality ofmobile units included in a frame picture at a time t in units of theblock, and obtaining motion vectors of said plurality of mobile units inunits of the block, in a case where identification code of saidplurality of mobile units included in a frame picture at a time (t−1)have been assigned in units of the block, and motion vectors of saidplurality of mobile units have been obtained in units of the block,wherein said picture processing device performs processing of: (a)moving a block j at the time (t−1), whose identification code is IDj andwhose motion vector is Vj, by the vector Vj to obtain a substantiallycorresponding block i at the time t and moving said block i by a vector−Vj to obtain a first box at the time (t−1), to calculate an evaluationvalue associated with a correlation between an image in said first boxat the time (t−1) and an image of said block i at the time t; (b) movinga block k at the time (t−1), whose identification code is IDk and whosemotion vector is Vk, by the vector Vk to obtain a substantiallycorresponding block which is said block i at the time t and moving saidblock i by a vector −Vk to obtain a second box at the time (t−1), tocalculate an evaluation value associated with a correlation between animage in said second box at the time (t−1) and said image of said blocki at the time t; and (c) assigning said IDj or said IDk to said block iat the time t on the basis of magnitudes of said evaluation valuescalculated in the processing (a) and (b).
 5. The mobile unitidentification apparatus of claim 4, wherein said evaluation value ofsaid processing (a) includes a value associated with an absolute valueof a difference between a motion vector of said block i at the time t onthe assumption that an identification code of said block i at the time tis IDj, wherein said evaluation value of said processing (b) includes avalue associated with an absolute value of a difference between a motionvector of said block i at the time t on the assumption that anidentification code of said block i at the time t is IDk and a motionvector of a block adjacent to said block i.